Immersive displays

ABSTRACT

A method of displaying images on an immersive display. The method includes receiving information from an external sensor or input device of the immersive display, based on the information received, detecting an object that conflicts with a virtual reality space, adjusting at least one dimension of virtual reality space to provide an adjusted virtual reality for display on the immersive display to accommodate for the object, and displaying the adjusted virtual reality on the display of the immersive display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 16/372,073, filed on Apr. 1, 2019, which is acontinuation of U.S. patent application Ser. No. 14/846,278, filed onSep. 4, 2015, (U.S. Pat. No. 10,248,285, issued on Apr. 2, 2019), whichclaims priority under 35 U.S.C. § 119(e) to provisional application Ser.No. 62/048,673, filed Sep. 10, 2014, which applications are expresslyincorporated by reference herein, in their entireties.

FIELD OF TECHNOLOGY

The present disclosure relates to immersive displays such asthree-dimensional (3D) displays for displaying virtual or augmentedreality environments.

BACKGROUND

Teleconferences and online events such as online meetings, classes, andconferences are useful, particularly in situations where factors such astime, cost, distance, or any combination of these factors make in-personmeetings difficult or impractical.

With increasing popularity of immersive displays, applications for suchdisplays are also increasing in number. Applications or uses forimmersive displays may involve simulation of real world environments andmay include multiple users that utilize immersive displays from variouslocations. For example, a boardroom or meeting room, a classroom, aconference room or any other room may be simulated for users to attend avirtual meeting or class. Attendees to such meetings, classes, orconferences may be represented by avatars and the attendees may interactwith each other by speaking, gesturing, or even writing notes for eachother.

Such displays are often head-mounted and in many cases block out some orall of the real environment around the user in order to immerse theuser, for example, in the virtual world. The virtual world oraugmented-reality is perceived by the user based on images displayedclose to the user's eyes.

Improvements in immersive displays and applications or uses of suchimmersive displays are desirable.

SUMMARY

According to one aspect, a method of displaying images on an immersivedisplay is provided. The method includes receiving information from anexternal sensor or input device of the immersive display, based on theinformation received, detecting an object that conflicts with a virtualreality space, adjusting at least one dimension of virtual reality spaceto provide an adjusted virtual reality for display on the immersivedisplay to accommodate for the object, and displaying the adjustedvirtual reality on the display of the immersive display.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the attached Figures, in which:

FIG. 1 is a system for providing a multi-user virtual event;

FIG. 2 is a simplified block diagram of an example of an immersivedisplay of the system of FIG. 1; and

FIG. 3 is a flowchart illustrating an example of a method of displayingimages on a display of an immersive display.

DETAILED DESCRIPTION

For simplicity and clarity of illustration, reference numerals may berepeated among the figures to indicate corresponding or analogouselements. Numerous details are set forth to provide an understanding ofthe examples described herein. The examples may be practiced withoutthese details. In other instances, well-known methods, procedures, andcomponents are not described in detail to avoid obscuring the examplesdescribed. The description is not to be considered as limited to thescope of the examples described herein.

The following describes a system and a method of operating the immersivedisplay. The method includes receiving from an external sensor or inputdevice of the immersive display, based on the information received,detecting an object that conflicts with a virtual reality space,adjusting at least one dimension of virtual reality space to provide anadjusted virtual reality for display on the immersive display toaccommodate for the object, and displaying the adjusted virtual realityon the display of the immersive display.

A system 100 for providing a multi-user virtual event is illustrated inFIG. 1. The system includes servers 102 that are coupled to a network104 or networks, which includes the internet and may optionally includea cellular network through which several client devices, nodes, orterminals may be connected. In the example of FIG. 1, five clientdevices are coupled to the network 104, including desktop computers 106,108, 110, a laptop computer 112 which is coupled to the networkwirelessly through a modem 114, and a smartphone 116. The servers 102store and execute software or firmware and communicate and cooperatewith software and firmware on the client devices 106, 108, 110, 112, 116via the network. The software and firmware on the client devices 106,108, 110, 112, 116 also communicate and cooperate with software andfirmware on respective immersive displays that may be worn by the users.

The servers 102, utilizing the software or firmware, provide the virtualenvironment, which may be a three-dimensional virtual environment, suchas a boardroom or meeting room, a classroom, a conference room or anyother room or other scene. The virtual environment provided is dependenton the type of event, the number of attendees, and any other suitablefactor or factors. An organizer or administrator may also choose fromdifferent possible environments.

The servers 102 also manage authorization of users via client devices tofacilitate attendance in the virtual environment by avatars representingthe users. The avatars enter the virtual environment to attend the eventsuch as a meeting, class, conference or other event.

The images, including the environment, are provided to the clientdevices 106, 108, 110, 112, 116 for display utilizing the immersivedisplays, which may be, for example, head-mounted displays worn by theusers.

A simplified block diagram of an example of an immersive display 200 isshown in FIG. 2. The immersive display 200 includes multiple components,such as a main processor 202 that controls the overall operation of theimmersive display 200.

The main processor 202 interacts with other components of the immersivedisplay 200, including, for example, a temporary storage device 204, amemory 206, a display device 208, a speaker 210, an auxiliaryinput/output (I/O) subsystem 212, external cameras 214, one or moreinternal cameras 216, one or more microphones 218, anorientation/movement sensor 220, one or more proximity sensors 222, acommunication subsystem 224, short-range communications 226, a powersource 228, and, optionally, other subsystems 230.

The temporary storage device 204 may be, for example, Random AccessMemory (RAM) that stores data that is processed by the main processor202. The memory 204, such as flash memory, is utilized for persistentstorage.

The immersive display 200 provides video output through the display 208,which includes an interface, a controller and at least one display todisplay images. The images displayed may be an image in front of atleast one of the user's eyes or may include a respective image in frontof each one of the user's eyes. In addition to the display 208, outputis provided via the speaker 210 or other audio output such as headphonesor earphones. The auxiliary input/output (I/O) subsystem 212 includes aninterface through which, for example, a USB controller or otherperipheral device may be connected.

Input to the immersive display may be provided via external sensors orinput devices such as the external cameras 214 mounted on the body ofthe immersive display 200. The external cameras 214 may include multiplecameras to obtain images extending around the user, i.e., 360° aroundthe user. The external cameras 214 may also include cameras to obtainimages in an upward direction from the user, and in a downward directionfrom the user. Each of the cameras includes the functional componentsfor operation of the camera, including the lens, the image sensor, and,optionally, a light sensor and light source, such as infrared lightemitting diodes (LEDs). Thus, the cameras provide images of the user'senvironment or surroundings. The cameras may be one or more of visuallight cameras, 3D sensing cameras, light field cameras, forward lookinginfrared cameras, near infrared cameras, ultraviolet cameras, or otherimaging devices.

The terms upward and downward are utilized herein to generally describedirection of view of the external cameras 214 relative to the user whenthe immersive display is worn by the user and the user is in an uprightposition, and such terms are not otherwise limiting.

The one or more internal cameras 216, referred to herein as the internalcamera 216, may be mounted on an inside of the body of the immersivedisplay and includes the functional components for operation of eachinternal camera, including the lens, the image sensor, and a lightsource, which may be a light source in the non-visible spectrum, such asinfrared LEDs. Although the interior of the immersive display 200 may bedark because exterior light is blocked out or partially blocked out, thelight source provides sufficient light for use of the internal camera216.

The one or more microphones, referred to herein as the microphone 218,may also be mounted in the body of the immersive display 200 to provideinput by converting audible information to electrical signals, which maybe processed by the main processor 202 and may be transmitted to anotherelectronic device to which the immersive display 200 is coupled. Forexample, the immersive display may be coupled to a smart-phone, a laptopcomputer, a tablet, a desktop computer, a game device, and any othersuitable electronic device.

The main processor 202 also receives signals from theorientation/movement sensor 220, which is coupled to the body of theimmersive display 200. The orientation/movement sensor may be, forexample, an accelerometer, a gyro sensor, or any other suitable sensoror combination of sensors that is or are utilized to detect direction ofmovement, direction of gravitational forces, and reaction forces so asto determine, for example, the orientation of the immersive display 200and the movement of the immersive display 200.

The one or more proximity sensors, referred to herein as the proximitysensors 222, may provide additional input to the main processor 202 todetect the presence of objects that are near or proximal to the sensorand thus to the user when the immersive display 200 is in use. Theproximity sensors 222 may be any suitable proximity sensors such as acapacitive or photoelectric proximity sensor.

The communication subsystem 224 receives signals from another electronicdevice such as the client devices 106, 108, 110, 112, 116 shown in FIG.1, and sends signals to the other electronic device to which theimmersive display is coupled. Thus, for example, the signals from themicrophone 218 or signals from the external cameras 216 or from theinternal camera 216 may be sent via the communication subsystem 224. Thecommunication subsystem 224 is also responsible for receiving signalsfrom the other electronic device for processing by the main processor202 to cause images, which may include video, to be displayed on thedisplay 208 and for audio to be output through the speaker 210.

The immersive display 200 optionally includes short-range communications226 to perform various communication functions. For example, theimmersive display 200 may include Bluetooth or infrared (IR)communications capability, for example, for communicating with aperipheral device or accessory.

The power source 228 may be one or more rechargeable batteries or a portto an external power supply to power the immersive display 200.

The systems and subsystems that interact with the main processor 202 andare described herein are provided as examples only. Other subsystems 230may also interact with the main processor 202.

Utilizing the images from the internal camera 216, the main processor202 may be operable to track eye motion. To track eye motion, the user'spupils may be tracked when the immersive display 200 is in use. The eyemotion tracking may also facilitate determination of what a user islooking at, for example, by triangulation to determine depth of anobject or image that a user is looking at. Alternatively, the internalcamera 216 may identify or track changes in muscles or muscle motionaround at least one of the user's eyes to identify movement of the eye,or may track changes in shape of a lens of an eye or changes in shape ofthe lens of each eye to identify a focal distance, facilitatingidentification of the depth of focus of a user.

Based on the eye motion tracking, the direction of a user's view, or thedirection that the user is looking, may be identified. The direction maybe, for example, an angle or angles, such as angular offset or offsetsfrom straight ahead. Thus, when a user glances upwardly, downwardly, orto either side, the direction is identified and the images displayedutilizing the display 208 may be changed or adjusted based on thedirection.

The main processor 202 is also operable to analyze the images from theinternal camera to track or identify a change in facial expression. Forexample, the main processor 202 may utilize primary facial featuretracking by tracking features such as lips, nose, and eyes.Alternatively, or in addition, movement of parts of the face may betracked. The main processor 202 may transmit facial expression data oran identification of the expression to the other electronic device towhich the immersive display 200 is coupled via the communicationsubsystem 222.

The main processor 202 is also operable to receive the image data fromthe external cameras and to transmit the data to the other electronicdevice, along with metadata for at least key frames for identifying theimage data such that the images can be stitched together to provideimages of the user's surroundings. Thus, the images from each of thecameras can be stitched together to obtain images of the user's entiresurroundings.

A flowchart illustrating a method of displaying images on a display ofan immersive display is shown in FIG. 3. The method may be carried outby software executed, for example, by one or more of the servers 102, byone or more of the client devices 106, 108, 110, 112, 116, by the mainprocessor 202 of the immersive display 200, or by a combination of thesedevices. Coding of software for carrying out such a method is within thescope of a person of ordinary skill in the art given the presentdescription. The method may contain additional or fewer processes thanshown and/or described, and may be performed in a different order.Computer-readable code executable by at least one processor to performthe method may be stored in a computer-readable medium, such as anon-transitory computer-readable medium.

A virtual reality space is identified at 302. The virtual reality spacemay be identified based, at least in part, on a host or user selectionvia an immersive display 200, or via a client device, such as one of theclient devices 106, 108, 110, 112, 116, coupled via the network 104 tothe servers 102. The virtual reality space may also be identified 302based on a type of meeting, conference, class, or other event identifiedby a host and, for example, a number of invitees to the event.

Information from the exterior of an immersive display 200 is received at304. The information may be, for example, obtained by the main processor202 of the immersive display 200 and sent, via one of the client devices106, 108, 110, 112, 116, to one or more of the servers 102. Theinformation may be information obtained utilizing the external cameras214 and the proximity sensors 222 of the immersive display 200. Thus,the information may include a location of an object, including distanceand direction relative to the immersive display 200.

In addition, to the information obtained using the external cameras 214,and the proximity sensors 222, the information may be informationobtained from, for example a peripheral accessory such as a glove, awatch, or a sock that is equipped with a proximity sensor or proximitysensors and is in communication with the immersive display 200 via theshort-range communications 226. Thus, gloves, a watch, or socks that maybe worn by the user to track movement of the hands or feet of the usermay also be utilized to provide proximity information to the mainprocessor 202. The gloves, watch, or socks may also be equipped withhaptic feedback devices such as piezoelectric actuators, or vibrationaldevices to provide the user with feedback based on the locations of theuser's hands or feet.

Based on the information obtained, including the images obtainedutilizing the external cameras 214 and the information obtained from theproximity sensors 222, objects that conflict with the virtual realityspace are identified at 306.

Thus, based on location of an object, including distance of the objectfrom the immersive display 100 or from the peripheral accessory anddirection of the object from the immersive display 100 or peripheralaccessory, objects that are located in the real environment of the userand that conflict with the virtual reality space identified at 302 areidentified at 306. The objects may include objects such as a table, achair, a wall, stairs, a post or pillar, or any other furniture orstructure. The objects may also include, for example, a robotic device,or a dog, a cat, a person, or other living being. Objects in the realenvironment may conflict with the virtual reality space when, based onthe location of the object and based on the virtual reality space, theobject would obstruct or inhibit the user from movement within thevirtual reality space. For example, an object in a location that wouldobstruct a user from moving down a hall or path in the virtual realityspace or from sitting in a location in the virtual reality spaceconflicts with the virtual reality space. Thus, predetermined regions inthe virtual reality space may be utilized to identify when objects inthe real space would interfere or obstruct a user from utilizing thepredetermined regions in the virtual reality space.

When no object conflicts with the virtual reality space at 308, themethod may continue at 304. Thus, the method may be carried continuouslyto identify objects that may conflict with the virtual reality space.For example, a user moves in real space, for example, down a hallway orinto a new room, objects in the hallway or new room that conflict withthe new virtual reality space are identified.

When one or more objects are identified, the method proceeds from 308 to310. To accommodate for the conflicting objects, adjustments or changesare identified to adjust the virtual reality that is displayed on thelocal display 208 of the immersive display 200. The adjustments orchanges include alterations to the dimensions of the virtual space thatis displayed such that the object no longer obstructs the alteredvirtual space. The alterations are identified by identifying adjustmentsto dimensions of the virtual reality space such that regions, which maybe predetermined regions, for user interaction such as free spaceintended for a user to walk in or through or intended for a user to sitin, correspond with or matches regions of the real space.

The adjustments that are identified may include a scalar adjustment thatis consistent or uniform across a linear dimension or may be adjustmentsthat are non-uniform such that a first portion of a length of a space isadjusted by an amount that differs from a second portion of the lengthof the space. The adjustments may also differ in each dimension.

For example, an adjustment may be identified to make a table in virtualspace larger, such that a table in the virtual space ends where thetable in the real space ends. By making the table in virtual spaceslightly larger, an area at the end of the table in real space maycorrespond with the area at the end of the table in virtual reality,facilitating use of the space by the user. In another example,adjustments may be identified to make the virtual space slightly smallerto fit the dimensions of a room in real space. According to anotherexample, adjustments may be identified to adjust the dimensions of adoorway in virtual reality to fit the dimensions of a door or hallway inreal space.

The images for displaying on the display 208 of the immersive display200 are adjusted at 312 based on the adjustments identified at 310. Theimages are adjusted for display on the immersive display 200 from whichthe information was obtained at 304. Thus, different adjustments may bemade for each user attending a meeting, conference, class or otherevent.

During the event, information that is transmitted from the immersivedisplay 200, via the client devices 106, 108, 110, 112, 116, to one ormore of the servers 102 is also adjusted by a reverse or an oppositechange at 314. For example, when a user looks at another user's avatarin virtual reality, the line of sight of the user is based on thevirtual space images displayed on the user's display. Because thevirtual space images are adjusted at 312, the direction that the user islooking, referred to as the direction of view of a user, for example, tolook at an avatar of another user, or to look at a virtual element suchas a virtual white board or other object, may not match the direction ofthe avatar or virtual element as displayed on a second user's immersivedisplay. Thus, the direction of view, or direction that a user islooking, is adjusted by a reverse or opposite change to accommodate forchanges made to the virtual reality space at 312 such that the seconduser can see what the first user is looking at. For such an adjustment,the direction that a first user is looking may be identified, forexample, based on the direction that the first user is facing andutilizing the internal camera 216 when the immersive display 200 is inuse to identify the line of sight. The direction may be, for example, anangle or angles, such as angular offset or offsets from the user. Thedirection that the user is looking may be transmitted to the servers 102and the direction is changed by an amount to account for the adjustmentsmade to the virtual reality space of the first user. Thus, a userlooking at a speaker in a virtual event, appears to all eventparticipants to be looking at the speaker.

In addition to changes to direction, changes may be made, for example,to distances that a user moves or changes to both direction anddistance. Thus, when a user moves to an area at an end of a virtualtable, the user's corresponding avatar appears to move to the area atthe end of the virtual table for all participants in the event.

In addition to changes to information transmitted from the immersivedisplay 200, information that is sent to the immersive display 200 isadjusted at 316 such that the information that is sent to the immersivedisplay is changed to accommodate for the changes made to the virtualreality space. For example, the direction of view of other participantsmay be adjusted such that another participant looking at the speakerappears to the user of the immersive display 200 for which adjustmentswere made to the virtual reality space at 312, to be looking at thespeaker. Similarly, other directions and distances may also be changedat 316 to accommodate for changes that are made to the virtual realityspace at 312.

When a change in the real space is detected based on informationreceived from the immersive display 200 at 318, the process continues at302. Thus, when a user moves to a different location or when a changeoccurs in the room in which a user is located, the process continues at302.

In one example, a user is one of four participants attending a virtualbrainstorming event. A virtual boardroom is identified at 302 based on ahost participant's selection and information regarding the realenvironment is obtained at 304 utilizing the external cameras 214 andthe proximity sensors 222. For the purpose of the present example, apillar conflicts with the virtual reality space because the pillar islocated in front of a virtual white board utilized in the virtual event.Absent an adjustment, the pillar would obstruct the user from writing onthe left side of the white board. The pillar is identified at 306 andthe process continues from 308 to 310 where adjustments are identified.An adjustment is identified at 310 in order to shift the virtual whiteboard to the right such that the pillar does not obstruct the user fromwriting on the left side of the white board. The adjustment is appliedat 312 and the virtual white board is shifted to the right for only theuser of the immersive display 200 in the room in which the pillar islocated. Other participants may be unaware of the shift or adjustmentsmade to the virtual reality space for the user. Information transmittedfrom the immersive display 200 is adjusted at 314 to accommodate forchanges made to the virtual reality space for the user. For example,when the user points to an element such as a drawing or part of adrawing on the virtual white board, the direction that the user's avatarpoints to is changed at 314 for other participants such that the otherparticipants can see and identify what the user is attempting to pointto. Similarly when another participant points to an element on thevirtual white board, the direction is adjusted at 316 for display on theuser's display 208.

The described embodiments are to be considered as illustrative and notrestrictive. The scope of the claims should not be limited by thepreferred embodiments set forth in the examples, but should be given thebroadest interpretation consistent with the description as a whole. Allchanges that come with meaning and range of equivalency of the claimsare to be embraced within their scope.

What is claimed is:
 1. An apparatus, comprising: an immersive display,displaying a virtual environment, operably coupled to a memory, and acomputing device; at least one detection device detecting the presenceof a real world object located in an apparently unobstructed locationwithin the virtual space; the immersive display programmed to modify thevirtual environment in response to the detecting by increasing at leastone dimension of an object within the virtual environment to make theobject an apparently obstructive virtual object in a place correspondingto the apparently unobstructed location of the real world object.
 2. Theapparatus of claim 1, the immersive display further coupled to acommunications device and further programmed to transmit themodifications to the virtual environment to at least one other secondaryimmersive display.
 3. The apparatus of claim 1, wherein the immersivedisplay detects movement of the real world object that obstructs thevirtual environment and shortens the at least one dimension of theobject within the virtual environment in response to the movement. 4.The apparatus of claim 1, comprising an internal camera coupled to thecomputing device to identify a direction of view of a user of theimmersive display.
 5. The apparatus of claim 4, wherein the immersivedisplay modifies the at least one dimension of the object at a time whenthe object is not in the direction of view of the user of the immersivedisplay.
 6. The apparatus according to claim 1, wherein the immersivedisplay accomplishes the detecting, at least in part, by calculatingwhether any objects in a real environment interfere with predeterminedregions of the virtual environment.
 7. The apparatus according to claim1, wherein the object within the virtual environment is furniture.
 8. Anapparatus, comprising: an immersive display operably coupled to amemory, a communication device, and a computing device; at least onedetection device detecting the presence of a real world object locatedat a place that corresponds to a location within the virtualenvironment; the immersive display operably coupled to one or moresecondary immersive displays; the immersive display and the one or moresecondary immersive displays rendering substantially a same virtualenvironment; the virtual environment containing an object (the“interactive object”) that a user of the immersive display and at leastone user of one secondary immersive display interact with; and theimmersive display detecting an obstruction that would impair theinteraction of the user of the immersive display with the interactiveobject, and in response to the detecting, moving the interactive objectso that the user of the immersive display may interact with the objectwithout the obstruction.
 9. The apparatus of claim 8, wherein the movingof the interactive object is not displayed on the one or more secondaryimmersive displays.
 10. The apparatus according to claim 8, wherein theimmersive display modifies at least one dimension of at least one othervirtual object in conjunction with the moving of the interactive object.11. The apparatus according to claim 10, wherein the modifications aredisplayed on the one or more secondary immersive displays.
 12. Theapparatus according to claim 8, wherein the moving of the interactiveobject is displayed on the one or more secondary immersive displays. 13.The apparatus according to claim 8, wherein the interactive object is avirtual shared workspace.
 14. The apparatus according to claim 13,wherein the virtual shared workspace is a white board.
 15. The apparatusaccording to claim 8, wherein an avatar representing the user of theimmersive display is modified on the one or more secondary displayswhile interacting with the interactive object so that it appears thatthe avatar is interacting with the interactive object in a location thatcorresponds to a location of the interactive object on the one or moresecondary displays.
 16. An apparatus, comprising: an immersive displayoperably coupled to a memory and a computing device, the computingdevice providing output for display on the immersive display; at leastone detection device capable of detecting the presence of a real worldobject located in an apparently unobstructed location within a virtualspace; the immersive display displaying a virtual environment; theapparatus altering an amount of movement within the virtual environmentthat corresponds to movement of an operator of the immersive displaysuch that the operator reaches a boundary within the virtual environmentbefore colliding with the real world object.
 17. The apparatus of claim16, where the altering is accompanied by a change in a length of atleast one dimension within the virtual environment.
 18. The apparatus ofclaim 16, where the altering is not accompanied by a change in a lengthof at least one dimension within the virtual environment.
 19. Theapparatus of claim 16, where the altering is reversed, at least in part,in response to movement of the real world object.
 20. The apparatus ofclaim 16, where the altering is displayed on at least one secondaryimmersive display.